NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning

Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. However, even fine-tuning the PLMs and doing inference are expensive, especially on edge devices with low computing power. Some general approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning, while very few one-shot compression techniques are explored. In this paper, we investigate the neural tangent kernel (NTK)--which reveals the gradient descent dynamics of neural networks--of the multilayer perceptrons (MLP) modules in a PLM and propose to coin a lightweight PLM through NTK-approximating MLP fusion. To achieve this, we reconsider the MLP as a bundle of sub-MLPs, and cluster them into a given number of centroids, which can then be restored as a compressed MLP and surprisingly shown to well approximate the NTK of the original PLM. Extensive experiments of PLM fine-tuning on both natural language understanding (NLU) and generation (NLG) tasks are provided to verify the effectiveness of the proposed method MLP fusion. Our code is available at https://github.com/weitianxin/MLP_Fusion.Comment: ICML 202

Protective effect of ethosuximide on hearing in NOD/LtJ mice via blockage of endogenous apoptosis

Purpose: To determine the protective effect of ethosuximide on the hearing of NOD/LtJ mice, and the underlying mechanism of action. Methods: The mice were randomly assigned to control and treatment groups (20 mice per group). Mice in the treatment group were administered ethosuximide intraperitoneally at a dose of 200mg/kg body weight (bwt), while those in the control group received an equivalent dose of saline via the same route. Both groups were subjected to auditory brainstem response (ABR) and distortion product otoacoustic emissions (DPOAE) tests, as well as determination of mRNA expressions of α1G, α1H, α1I, m-calpain, μ-calpain, and caspase-3. Results: At ages of 6 and 9 weeks, ABR values were significantly lower in the treatment group than those in the control group (p < 0.05). At age 3, 6 and 9 weeks, control group DPOAE values were much lower than those in the treatment group. However, at signal frequency of 35344 Hz, DPOAE value was significantly reduced in the treatment group (p < 0.05). There was significant down-regulation in mRNA expressions of α1G, α1H, α1I, m-calpain, μ-calpain and caspase-3, in the treatment group, when compared with the control group (p < 0.05). Conclusion: Ethosuximide delays mice hearing loss and protects their hearing via a mechanism involving blockage of endogenous apoptotic pathways. This mechanism may provide guidance in the search for suitable new drugs. Keywords: Ethosuximide, Endogenous apoptosis, Hearing, Protectio

Bibliometric and visualization analysis of research trend in mental health problems of children and adolescents during the COVID-19 pandemic

ObjectivesTo analyze the evolution of research on children and adolescents mental health issues during COVID-19 pandemic and discuss research hotspots and cutting-edge developments.MethodsThe literature obtained from the web of science core collection as of June 28, 2022, was analyzed using Citespace, VOSviewer bibliometric visualization mapping software.ResultsA total of 6,039 relevant papers were found, of which 5,594 were included in the study. The number of literatures is growing since 2020; and the country, institution, and journal publications were analyzed. The co-citation analysis shows that there are more research articles among the highly cited articles and a lack of systematic reviews that use critical thinking for review. In the cluster analysis, mental health and life change were the most representative. The timeline view of the keywords shows that Online learning (#0), Public health (#1), and Mental health (#2) are the three largest clusters and shows the change over time.ConclusionThis study helped analyze the mental health of children and adolescents during the COVID-19 pandemic and identified hot trends and shortcomings, which are important references for the theoretical basis of future research and decision making and technical guidance for systematic reviews

Enhancing Mixup-Based Graph Learning for Language Processing via Hybrid Pooling

Graph neural networks (GNNs) have recently been popular in natural language and programming language processing, particularly in text and source code classification. Graph pooling which processes node representation into the entire graph representation, which can be used for multiple downstream tasks, e.g., graph classification, is a crucial component of GNNs. Recently, to enhance graph learning, Manifold Mixup, a data augmentation strategy that mixes the graph data vector after the pooling layer, has been introduced. However, since there are a series of graph pooling methods, how they affect the effectiveness of such a Mixup approach is unclear. In this paper, we take the first step to explore the influence of graph pooling methods on the effectiveness of the Mixup-based data augmentation approach. Specifically, 9 types of hybrid pooling methods are considered in the study, e.g., $\mathcal{M}_{sum}(\mathcal{P}_{att},\mathcal{P}_{max})$. The experimental results on both natural language datasets (Gossipcop, Politifact) and programming language datasets (Java250, Python800) demonstrate that hybrid pooling methods are more suitable for Mixup than the standard max pooling and the state-of-the-art graph multiset transformer (GMT) pooling, in terms of metric accuracy and robustness

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and improving state-of-the-art features for long-range contact prediction

Multimodal evaluation of the bloodstream alteration before and after combined revascularization for Moyamoya disease

ObjectiveThis study aimed to explore the hemodynamic changes before and after anastomosis in patients with Moyamoya disease (MMD) using multiple models.MethodsWe prospectively enrolled 42 MMD patients who underwent combined revascularization. Intraoperative FLOW800 was performed before and after anastomosis, and parameters was collected, including maximum intensity, delay time, rise time, slope, blood flow index, and microvascular transit time (MVTT). Additionally, preoperative and postoperative hemodynamic parameters were measured using color Doppler ultrasonography (CDUS), including peak systolic velocity, end-diastolic velocity, resistance index (RI), pulsatility index (PI), and flow volume. Subsequently, the correlation between FLOW800 and CDUS parameters was explored.ResultsA total of 42 participants took part with an average age of 46.5 years, consisting of 19 men and 23 women. The analysis of FLOW800 indicated that both the delay time and rise time experienced a substantial decrease in both the recipient artery and vein. Additionally, the MVTT was found to be significantly reduced after the surgery (5.7 ± 2.2 s vs. 4.9 ± 1.6, p = 0.021). However, no statistically significant differences were observed among the other parameters. Similarly, all postoperative parameters in CDUS hemodynamics exhibited significant alterations in comparison to the preoperative values. The correlation analysis between FLOW800 and CDUS parameters indicated a significant association between MVTT and RI and PI, no significant relationships were found among the other parameters in the two groups.ConclusionThe hemodynamic outcomes of the donor and recipient arteries demonstrated significant changes following bypass surgery. The parameter of time appears to be more precise and sensitive in assessing hemodynamics using FLOW800. Multiple evaluations of hemodynamics could offer substantial evidence for perioperative management

Factors associated with distant metastasis in pediatric thyroid cancer: evaluation of the SEER database

Objectives: Controversies regarding factors associated with distant metastasis in pediatric thyroid cancer remain among the scientific community. The aim of this study was to investigate factors influencing distant metastasis in pediatric thyroid cancer. Methods: We reviewed 1376 patients (aged 2 to 18 years) with thyroid cancer treated between 2003 and 2014. Data collected and analyzed included sex, race, age at diagnosis, year of diagnosis, pathological type, number of tumor foci, tumor extension, T-stage, N-stage, surgical procedure and radiation. Univariate and multivariate analyses were conducted to evaluate factors influencing distant metastasis of pediatric thyroid cancer. Results: In the univariate analysis, factors influencing distant metastasis of thyroid cancer were age at diagnosis (P 0.05). Furthermore, according to chi-squared test, younger pediatric thyroid cancer patients with higher T- and N-stages are more likely to have distant metastasis. Conclusion: Age at diagnosis, T-stage and N-stage influence distant metastasis of thyroid cancer patients aged 2 to 18 years; accordingly, more radical treatments may need to be used for patients with those risk elements

Solvent engineering of oxygen-enriched carbon dots for efficient electrochemical hydrogen peroxide production

The development of cost-effective and reliable metal-free carbon-based electrocatalysts has gained significant attention for electrochemical hydrogen peroxide (H2 O2 ) generation through a two-electron oxygen reduction reaction. In this study, a scalable solvent engineering strategy is employed to fabricate oxygen-doped carbon dots (O-CDs) that exhibit excellent performance as electrocatalysts. By adjusting the ratio of ethanol and acetone solvents during the synthesis, the surface electronic structure of the resulting O-CDs can be systematically tuned. The amount of edge active CO group was strongly correlated with the selectivity and activity of the O-CDs. The optimum O-CDs-3 exhibited extraordinary H2 O2 selectivity of up to 96.55% (n = 2.06) at 0.65 V (vs RHE) and achieved a remarkably low Tafel plot of 64.8 mV dec-1 . Furthermore, the realistic H2 O2 productivity yield of flow cell is measured to be as high as 111.18 mg h-1  cm-2 for a duration of 10 h. The findings highlight the potential of universal solvent engineering approach for enabling the development of carbon-based electrocatalytic materials with improved performance. Further studies will be undertaken to explore the practical implications of the findings for advancing the field of carbon-based electrocatalysis.The project was funded by the Shanghai Pujiang Program (21PJD022), the National Natural Science Foundation of China (No. 21901154)

New Research on MEMS Acoustic Vector Sensors Used in Pipeline Ground Markers

According to the demands of current pipeline detection systems, the above-ground marker (AGM) system based on sound detection principle has been a major development trend in pipeline technology. A novel MEMS acoustic vector sensor for AGM systems which has advantages of high sensitivity, high signal-to-noise ratio (SNR), and good low frequency performance has been put forward. Firstly, it is presented that the frequency of the detected sound signal is concentrated in a lower frequency range, and the sound attenuation is relatively low in soil. Secondly, the MEMS acoustic vector sensor structure and basic principles are introduced. Finally, experimental tests are conducted and the results show that in the range of 0°~90°, when r = 5 m, the proposed MEMS acoustic vector sensor can effectively detect sound signals in soil. The measurement errors of all angles are less than 5°

Spatially Non-Stationary Relationships between Changing Environment and Water Yield Services in Watersheds of China’s Climate Transition Zones

Identifying the spatial and temporal heterogeneity of water-related ecosystem services and the mechanisms influencing them is essential for optimizing ecosystem governance and maintaining watershed sustainable development. However, the complex and undiscovered interplay between human activities and natural factors underpins the solutions to the water scarcity and flooding challenges faced by climate transition zone basins. This study used a multiple spatial-scale analysis to: (i) quantify the spatial and temporal variations of the water yield ecosystem service (WYs) of the Wei River Basin (WRB) from 2000 to 2020 using the InVEST model and remote sensing data; and (ii) look at how human activities, climate, topography, and vegetation affect the WYs at the climate transition zone sub-catchment scale using the geographical detector model and multi-scale geographically weighted regression (MGWR). The conclusive research reveals that there would be a gradual increase in WYs between the years 2000 and 2020, as well as a distinct and very different spatial aggregation along the climatic divide. The average yearly precipitation was shown to be particularly linked to the water yield of the WRB. The interplay of human, climatic, plant, and terrain variables has a substantially higher influence than most single factors on the geographical differentiation of WYs. Bivariate enhancement and non-linear enhancement are the most common types of factor interactions. This shows that there are significant interactions between natural and human variables. Our study shows that precipitation and temperature are the main factors that cause WYs in the semi-arid zone. In the semi-humid zone, precipitation and vegetation are the key controlling factors that cause WYs. We provide new perspectives for understanding and optimizing ecosystem management by comparing the drivers of WYS in sub-basins with different climatic conditions. Based on the findings, we recommend that particular attention should be paid to ecosystem restoration practices in watersheds in climatic transition zones